Does Writing Improve as a Function of Number of Reviewers?

Evaluation management systems, especially reciprocal peer evaluation systems, often have an assumption that more reviewers will produce better results. This tendency is labeled as the maxima strategy. This study examines the maxima strategy from both agreement and performance perspectives with the intent of examining the role of information in reliability and performance through an optimal number of peer reviewers per evaluate in writing. It was found that the maxima strategy works consistently with agreement perspectives, whilst the relationship between the maxima strategy and performance improvement follows an inverted U-shaped function. Accordingly, we recommend that the number of reviewers needs to be decided on the optimal balance between reliability and performance, which maximizes writers’ performances without sacrificing evaluation reliability. Consistent with a recent movement of integrating evaluation assessment and training (Dochy, Segers, & Sluijsmans, 1999) with evaluation management systems, reciprocal peer evaluation (RPE) and its distinction of relying upon multiple reviewers has gained popularity throughout education and training (Magin, 2001) and in numerous organizations (Harris & Schubroeck, 1988; Illgen, 1999; Katzenbach & Smith, 1993). According to the American Society for Training and Development (ASTD), for instance, about 33% of firms utilized RPE systems in 1999, an increase from only 10% in 1997 (ASTD, 1999). Unlike typical expert-based evaluation management systems where participants receive evaluations only from experts, participants in RPE systems maximize resources by playing dual roles: reviewer and writer. As a reviewer, each participant provides peer feedback. As a writer, each receives feedback from peers. Thus, RPE systems allow participants to construct as well as receive evaluations. In this study, we examine from assessment perspectives and also from performance perspectives the optimal number of peer reviewers for effective evaluation management in RPE systems. Considering that a primary advantage of RPE systems is providing multiple peer reviewers and hence more feedback, deducing the optimal number of reviewers warrants examination. However, few empirical studies have systematically examined this issue. Therefore, the question of optimal number of reviewers is still open to examination. In this situation, prevalently accepted is what has come to be known as the maxima strategy, meaning the implicit assumption that more reviewers will produce better results. The maxima strategy provides RPE systems with various advantages not afforded by traditional expert-based evaluation management systems. For example, RPE system participants receive rich feedback without sacrificing expert resources (Cho & Schunn, in press). Large numbers of reviewers provide more information about writers’ problems (Wittenbaum & Stasser, 1996). Also, participants generate as well as receive evaluations, which may help participants actively reflect upon their own performance as well as that of others (Schriver, 1990). They develop crucial evaluation skills applied to their professions (Oldfield & MacAlphine, 1995) and in the process dispel negative connotations about evaluation. In addition, their participation motivates them to engage more fully with their tasks (Michaelson & Black, 1984). Despite advantages, three major concerns discourage using RPE systems in practice: reliability, outliers, and performance. The reliability and outlier concerns are addressed from the assessment perspective, while the performance concern is addressed from the learning and performance perspective. Interestingly, all three concerns are typically addressed by using the maxima strategy in RPE systems. Reliability Perspective Reliability is often measured as consistency which concerns the degree to which different peer reviewers generate consistent evaluations on the same tasks. Various studies reported medium or low reliabilities among peer reviewers (e.g. Mowl & Pain, 1995), while some studies report high reliability (e.g. Hughes & Large, 1993). What these studies claim to measure as reliability is actually mean reliability, defined as expected reliability of an individual reviewer (Rosenthal & Rosnow, 1991). For example, when the mean reliability between two reviewers is .4, it indicates the expected reliability of either single reviewer, not that of combined reviewers. Therefore, what this study needs to know is the aggregate reliability of the total reviewers, known as effective reliability (Rosenthal & Rosnow, 1991). Effective reliability provides the measure of composite internal consistency of combined reviewers. To compute this reliability, we use the following formula adapted from the Spearman-Brown formula (Rosenthal & Rosnow, 1991): r n r n R ) 1 ( 1 − + = where R is the effective reliability coefficient, n is the number of reviewers, and r is the mean reliability among reviewers. Therefore, effective reliability will demonstrate an increasing asymptote function as the number of reviewers increases (see Figure 1a). Another issue discouraging RPE system use is outliers due to evaluation biases, particularly when participants’ identities are disclosed. Biases cause unfair peer evaluations (Bence & Oppenheim, 2004; Michaelson & Black, 1994) to which writers are generally sensitized (Michaelson & Black,